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• SECTION I 
INTRODUCTION 



Found within most districts which have special programs of one 
kind or another is a person referred to as a director, specialist, 
consultant, or coordinator^-in' this case, resource teacher. This 
person, regardless of title, is responsible for the implementation 
and evaluation of the program in which he/she is involved. In the 
past, concerns have been expressed in regard to the appropriateness 
of the same individual being charged with the implementation of a 
program and also its evaluation. In many cases, this concern is 
probably justified; yet, school districts are reluctant to spend the 
thousands of dollars necessary to buy an outside evaluation or employ 
a permanent district evaluator. Tlie answer to 'the dilemma is to go to 
the Los Angeles County Superintendent of Schools Office and arrange for 
the services of one of their program a\.:ditors in the Division of Program 
Evaluation, Research, and Pupil Services to audit the program. 

The actual procedures and steps to develop the audit plan and its 
implementation are outlined in Section III of this article. The 
important point to have in mind at this time is to understand who 
is actively involved in a program audit and what their roles are. 
In summary, the personnel involved are as follows: 

District Employee : usually the coordinator, consultant, 
resource teacher, etc., responsible for implementing the 
program. This person acts as the evaluator. 

County Office Consultant : the person designated as the 
program auditor. This person assesses and reports the 
degree of his agreement with the evaluation report regarding 
the level of attainment of pre-specif ied program objectives 
as measured by audit sample data analysis. This auditor 
prepares the interim and final audit reports which are 
considered confidential and go to the superintendent of the 
district requesting the program audit. 
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SECTION II 

AN OVERVIEW OF THE EDUCATIONAL PROGRAM AUDIT PROCESS 



Over the last few years, that is during the pilot testing and full 
implementation of the audit services on a free to requesting district 
and community college basis, several questions have been routinely 
raisedr This section will present an overview of the educational 
• program audit process by way of the question and answer format, using 
questions most frequently asked. 

WFIAT IS AN EDUCATIONAL PROGRAM AUDIT? 

DEFINITION: Aa ^educational program audit is a .performance control 
process based upon external reviews conducted by qualified outside 
consultants. It is designed to verify the results of the evaluation 
of an educational program and to a'ssess the appropriateness of evalu- 
ation procedures used for determining the effectiveness of the opera- 
tion and management of the program. 



WHY HAVE AN AUDIT? WHAT MIGHT RESULT FROM IT? 

There are many reasons why local educational agencies request program 
audits. Some of the results of past audits are: 

1. The identification of clearer definitions of program personnel 
responsibilities. 

2. The gathering of needed baseline data on the level of success . 
of an educational program. 

3. The determination of test scoring discrepancies. 

4. The identification of model or "turnkey** programs which deserve 
to be developed and expanded. 

5. The identification of areas for inscrvicc training. 

6. The enhancing of credibility for the program and for the local 
educational agency. 
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WHAT ARE THE PROFESSIONAL ROLES OF THE EDUCATIONAL PROGRAM AUDITOR? 

The educational program auditor brings a relatively new professional 
role to education. He serves as a third party, free of local ties 
and interests, -who verifies the results of the internal evaluation 
of an educational program. 



THE AUDITOR IS: 

. K Independent from the program to be audited. 
2. A reviewer and not a decision-maker , 
3» A reporter and an observer. 

4. A professional who exercises care and integrity in 
performing the audit examination and in preparing audit 
reports. 

5. Equipped to innovate within his own profession and to 
encourage reforms in the schools. 



WHAT -ARE THE STAGES IN THE AUDIT PROCESS? 

STEP A: Of uional Pre-Audit Assistance: 

Educational Program Evaluation Assistance 

Local educational agencies may elect to receive Educational Program 
EvaluaJtaon Assistance prior to havings an audit of.,a program. Educational 
Program Evaluation Assistance is a service whi'Ch ii intended to strengthen 
the internal process of evaluation in a local educational agency. In 
collaboration with a county consultant, the local evaluator designs an 
effective evaluation sy'g^em with appropriate statistical procedures which 
includes development of performance objectives, process objectives, evalu- 
ation specifications, the evaluation design, and related documents. 

Step 1 - Orientation . 

Orientation sessions to the audit may be conducted for local 
educationnl agency personnel. 

Step 2 - Review of the Total Program 

The product and process objectives, the evaluation specifications, 
the design management plan, time frame, and related documentation 
are reviewed to ensure the auditability of the program. 
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Step 5 ~ Development of the Audit Plan 

The audit plan is a planning and operational control document for 
the auditor and a quality and management control document for both 
the local program director and the local program evaluator. This 
plan indicates the technique*;, schedules, processes, and procedures 
which the auditor will use in judging the adequacy of the evaluation 
process and in verifying 'the evaluation findings. One method of 
random sampling frequently used is matrix sampling. 

Step 4 - The Audit Contract 

The audit contract is a written agreement between the local education 
agency and the Los Angeles County Superintendent of Schools Office. 
The purpose of the contract is to make explicit the conditions of the 
audit agreement. One section, for example, is the Assurance of 
Confidentiality section. 

Step 5 ~ On-Site Visit (s) 

Typically, the auditor makes one announced on-site visit and one 
unannounced on-^site visit. During on-site visits, the auditor 
collects firsthand data as identified in the audit plan and for 
the purpose of ascertaining the degree of attainment of process 
objectives. 

Step 6 - Interim and Final Audit Reports 

Typically, the -auditor presents an interim report following each 
on-site visit and a final audit report following the receipt of the 
final evaluation report. The final audit report is presented to the 
program evaluator, program director, and superintendent, usually 
within twenty working days of receipt of the final evaluation report. 
All of the reports are confidential and presented to only the program 
evaluator, program director, and superintendent. 

Step 7 - Exit Interview and Local Educational Agency Evaluation 
of the Educational Program Audit Service they have Received 



It is recognized that the audit process should itself be evaluated 
by those who receive the service. 



CAN LOCAL EVALUATORS RECEIVE ASSISTANCE? 

Yes. Should a local evaluator of a program which is, being audited desire 
consultant assistanct^ in the development of an evaluation management plan, 
statistical data analysis, data synthesis, evaluation report writing or' 
interpretation, he should, as soon as possible, initiate a request for 
assistance by writing to Dr. Gordon Footman, Director, ' Division of Program 
Evaluation, Research, and Pupil Services, Los Angeles County Superintendent 
of Schools Office, 9300 East Imperial Highway, Downey, California 90242. 
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CAN PROGRAM OBJECTIVES BE REVISED? 

Yes, upon the agreement between the local j^rogram personnel and the 
auditor. . 



WHAT AUDIT TRAIL DOCUMENTS SH.OULD PROGRAM PERSONNEL KEEP? 

In order to do the. job of verifying the evaluation of an educational 
program, the auditor must have direct access to firsthand evaluation 
data. ^The rule here is '^Kccp all evaluation trail documents. Keep 
whatever measure is used by the evaluator to determine whether or not 
each program objective is attained." Evaluation trail documents often 
used are test bookelts and answer sheets, pre- to post- tests, or 
post- tests only, summary sheets, levels mastered on criterion referenced 
tests, and attendance rosters for inservice staff meetings. 



HOW DOES THE AUDIT RELATE TO THE STULL BILL? ' 

Whereas the Stull Bill is concerned with evaluation of a single person, 
the audit is concerned with verifying the evaluation of an educational 
program. The local program evaluator will present a final evaluation 
report, and possibly interim reports as well, on the attainment or 
nonattainment of program objectives. As an outside, thir.d party, the 
auditor samples evaluation data and verifies the results of the internally 
conducted program evaluation. The auditor's reports include audit findings 
relative to the attainment or nonattainment of program objectives. While 
the Stull Bill is concerned with a single person, the audit is concerned 
with a program. No person is mentioned by name in an audit report. If a 
program process objective were stated as "All instructors teaching a certain 
level will receive the math textbook. Mathematics Fundamentals , by the first 
day of class," and if during the on-site visit the auditor found that four 
instructors out of ten interviewed did not receive the books by the date 
indicated, the audit report would present just this information. The .point 
of focus for the audit is the program. 



WHAT IF OTHER QUESTIONS COME UP? 

Call your auditor of record to^di'scuss the situation. 
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SECTION III 
PLANNING AND IMPLEMENTING A PROGRAM AUDIT 



The following diagram represents the steps followed by the Rowland 
Unified School District in developing a program audit plan for their 
mentaHy gifted minor program. 



April: District personnel 
writes application for and 



Program Evaluation, Research, 
and Pupil Services Division, 
County Office of Education 



June: Meetings held between 
County and District Personnel 
and rough draft of Audit Plan 
written (plan based upon State 
Application) 

' V 



July: Proposed Audit Plan 
review and timeline proposed 

i 



September: Final Audit Plan 
Revision meeting held to establish 
dates, visitations , number of 
reports, etc. 








October: June Audit Plan imple- 
mented and process put into motion- 
Reports and recommendations made by 
District Evaluator and Auditor. 



June-July : Revise Audit Plan for 
following school year, 



gains approval trom btate 
for continuing MGM Program 

I 

May: District decides to 
develop a Program Audit Plan 



The foregoing procedures are not meant to be the rule of procedure 

but simply a planning guide. Each district varies and should establish 

its own organizational and procedural plan. 
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SECTION IV 

SOME RESULTS OF AND LESSONS LEARNED DURING THE 

EVALUATION AND PROGRAM AUDIT OF THE ROWLAND 
UNIFIED. MENTALLY GIFTED MINORS PROGRAM-. -1974-75 



Part A: Timing of the on-site visits--the evaluator should do his 

on-sito visits before the auditor begins his on-site visits 

Audit and evaluation consultants from the Los Angeles County have 
recommended that an on-site visit- and an evaluation report by the 
internal evaluator precede the on-site visit and audit report by 
the external program auditor. In this way, audit consultants 
have resisted "being the first one on the scene." However, the 
audit plan for the Rowland MGM Program called for an on-site visit 
by the auditor on November 18 and 19, with an audit report due in 
December. At the same time, it called for an evaluation report 
much later than the auditor's visit. This is now seen as an unfortu- 
nate error. The-^evaluator should have visited first; the evaluator 's 
report should have been shared by^ program managers and teachers before 
the auditor appeared on-site . 

The guideline that "the evaluator should be the first one on the 
scene" is reinforced in more concrete terms by an illustration of 
what happened relative to Objective 8.0 of the Rowland MGM Program 
Audit Plan. Objective 8.0 stated, "Each MGM studenX^will create a 
student log no later than November 15, 1974." ^ When the auditor arrived 
on November 18, the evaluator was only half through with his on-site 
visits to the schools. Since the evaluation report was not due until 
January 31, 1975, the evaluator seemed to have plenty of time to do 
his on-site visits. 

Table I shows when the evaluator visited the twenty schools and what he 
found relative to Objective 8.0, 
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. ..TABLE I 


Column A 


- Date 


of Evaluator' 


s visitation 


Column. B 


- X indicates 


logs had 


been 


I made 


Column C 


- X indicates 


logs had 


not 


been made 


, Column D 


- Date 


of Auditor's 


visitation 


Column R 


- X indicates 


logs had 


been 


made 


Column F 


- X indicates 


logs had 


not 


been made 




ScKbol 

• 


A 


B C 


7 

D 


E 


F 


Comments 






Alvarado 


11/18 


X 


11/18 


X 




Two visits made to 


















cover all classrooms 


















needing checking 






Blandford 


11/18 


X 


11/18 


X 










Farjardo 


11/20 


X . 














Giano 


11/19 


X 


11/19 


X 










Hollingworth 


11/18 


X 


11/19 


X 










Hurley 


11/29 


X 














Jellick 


11/20 


X 


11/18 


X 










Killian 


11/19 




11/17 




X 


Ki llian logs were 








11/25 


X 








checked twice and 


















were made at time 






* 












of second visit 






La Seda 


11/26 


X-; 














Nogales High School 


11/17 


X ' 














Northam 


11/26 


X 














Rincon 


11/25 


X 














Rorimer 


11/26 


X 








One teacher had logs 


















ready - one teacher 


















did not have logs 


















ready 






Rowland Elementary- 


11/19 


X 


11/18 


X 










Rowland High School; 


11/19 


X 


•11/17 


X 










Shelyn 


11/21 


X 














Villacorta 


11/28 


X 


11/19 


X 










Ytjarra 


11/20 


X 


11/19 


X 










Yorbita 


11/27 


_x _ 














TOTALS 


20 


11 i 


10 


9 


1^ 
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As the first on-site visitor. to some program schools out of the 
seven K-6, two 7-8, and one high school visited, the auditor (after 
talking with seventeen teachers and reviewing data on thirty-two MGM 
students) was forced to report that Objective 8.0 was not attained. 
That conclusion was reached since the auditor found one school out of-v • 
the ten at which student logs had not yet been created. The logs were 
to have been created by November 15, as noted in the objective; but by 
November 18 aiid 19, one school- out of ten had not seen the creation of 
the student logs. The auditor did report that student logs had been 
crfeated at nine of the ten schools and that at seven of the schools 
MGM students had already logged' such items as a field trip, a career 
to which they had been exposed, a speaker they had heard, or a creative 
product they tliemselves had created. 

The auditor also made the following comments regarding possible revision 
of the evaluation plan: 

The auditor views the Evaluation Plan as satisfactory but 
possible to improve. Accordingly, the auditor recommends 
that the program evaluator make a series of on-site visits 
three to six weeks prior to the unannounced second on-site 
visit of the auditor (March 25, 1975). The auditor also 
recommends -th^t the evalUcitor, after the evaluator' s on-site 
visits- communicate in summary form the results of his 
internal evalua^pn to program personnel at each school site. 
While this actfvity was not included in the aud;Lt plan for 
the Rowland MGM program . audit , the auditor believes that the 
implementation of this activity would provide the coordinator/ 
evaluator with another ''bench-mark'* evaluation and an opportu- 
nity to make any necessary adjustments in program operation 
before the external auditor arrives (approximately Marcli 25, 
1975). 

But, in a sense, it was already too late; the error had been made. 
An "outside person," the external auditor,- had "come to our school 
first" and in checking on the statii^ of the program had found that, 
at one school, student logs had not been created on time. Unfortunately, 
this is what happened. What should have happened was something like 
the following. First, all program participants, including MGM teachers, 
would have directly or indirectly helpied develop the objectives, manage- 
ment plan and time frame included in the evaluation and audit plans. 
Therefore, nil participants would be fully aware of the deadlines, 
SccoirI, the first person to check on the a^ttainmcrit or nonattainmcnt 
of program objectives should be the internal evaluator. Then when he 
writes, delivers, and discusses the evaluation report with program 
participants, he would also recommend increased effort to. attain 
objectives that he found had not been attained. At that point, after 
an "internal and friendly" appraisal had been made and discussed, the 
program managers could invoke the fact that an "outside" auditor will 
be coming to schools^ in a few days or weeks to provide 'a second check 
as to the attainment or nonattainment of objectives. 
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This latter procedure is viewed. to be the superior one since it provides 
not only for the opportunity for a maximum amount of internal monitoring, 
but also for team effort to occur prior to the on-site visits of the 
external auditor. In a sense, in the ideal situation, the internal 
evaluation can serve to motivate participants toward focused areas that 
need improvemcnt--just in time for that other on-site visit--this time 
by the auditor. 



Part B: Percentages--Look out for them 

ft: 

In one case wc used a figure of 90% when only twD people were 
involved in population to be checked. 



Dates-'Use a district m aster calendar to plan the auditor's 
visits — 



We scheduled the auditor's second visitation during the spring vacation. 

State and district report dates vary and some flexibility for this should 
be built into the audit plan. 



Part D: Class Records--Use a standardized or semi-standardized form 

If the auditor ^' s to check classroom records (in this case student logs) 
a standardized or semi-standardized form' should be used to insure all 
information is recoi*ded. 
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SECTION V 
ADVANTAGES OF A PROGRAM AUDIT 



The following are some advantages realized by the Rowland Unified 
School District in regard to the developing and implementing of a 
program audit plan. 

1. Cost: * For no cost, a district can obtain the services 
of an external program expert to validate the evaluation 
of a particular program. 

2. Credibility: A program audit lends credibility to 
in-district evaluations of ongoing programs. 

3. Influence: By informing, "line" administrative personnel 
(principal) of a program audit, a staff person (coordinator) 
gains influence at building site level in terms of insuring 
that the program is going well, 

4. Staff: Teachers involved in a program which is being audited 
tend to be more concerned with meeting objective deadlines, 

5. Program Improvement: The use of an audit plan in the district 
enables district personnel to more readily identify areas of 
program needs .and improvements. It should be kept in mind 

that program audits are audits of a program and not of personnel 
or fujids. 
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